Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations.

Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1-4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models' predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models' forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models' past predictive performance. Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models' forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models' forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models' forecasts of deaths (N=763 predictions from 20 models). Across a 1-4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks. Funding: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

Impact of SARS-CoV-2 vaccination of children ages 5-11 years on COVID-19 disease burden and resilience to new variants in the United States, November 2021-March 2022: A multi-model study.

Background: The COVID-19 Scenario Modeling Hub convened nine modeling teams to project the impact of expanding SARS-CoV-2 vaccination to children aged 5-11 years on COVID-19 burden and resilience against variant strains. Methods: Teams contributed state- and national-level weekly projections of cases, hospitalizations, and deaths in the United States from September 12, 2021 to March 12, 2022. Four scenarios covered all combinations of 1) vaccination (or not) of children aged 5-11 years (starting November 1, 2021), and 2) emergence (or not) of a variant more transmissible than the Delta variant (emerging November 15, 2021). Individual team projections were linearly pooled. The effect of childhood vaccination on overall and age-specific outcomes was estimated using meta-analyses. Findings: Assuming that a new variant would not emerge, all-age COVID-19 outcomes were projected to decrease nationally through mid-March 2022. In this setting, vaccination of children 5-11 years old was associated with reductions in projections for all-age cumulative cases (7.2%, mean incidence ratio [IR] 0.928, 95% confidence interval [CI] 0.880-0.977), hospitalizations (8.7%, mean IR 0.913, 95% CI 0.834-0.992), and deaths (9.2%, mean IR 0.908, 95% CI 0.797-1.020) compared with scenarios without childhood vaccination. Vaccine benefits increased for scenarios including a hypothesized more transmissible variant, assuming similar vaccine effectiveness. Projected relative reductions in cumulative outcomes were larger for children than for the entire population. State-level variation was observed. Interpretation: Given the scenario assumptions (defined before the emergence of Omicron), expanding vaccination to children 5-11 years old would provide measurable direct benefits, as well as indirect benefits to the all-age U.S. population, including resilience to more transmissible variants. Funding: Various (see acknowledgments).

Evaluating epidemic forecasts in an interval format.

For practical reasons, many forecasts of case, hospitalization, and death counts in the context of the current Coronavirus Disease 2019 (COVID-19) pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub (https://covid19forecasthub.org/). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.

An evaluation of prospective COVID-19 modelling studies in the USA: from data to science translation.

Infectious disease modelling can serve as a powerful tool for situational awareness and decision support for policy makers. However, COVID-19 modelling efforts faced many challenges, from poor data quality to changing policy and human behaviour. To extract practical insight from the large body of COVID-19 modelling literature available, we provide a narrative review with a systematic approach that quantitatively assessed prospective, data-driven modelling studies of COVID-19 in the USA. We analysed 136 papers, and focused on the aspects of models that are essential for decision makers. We have documented the forecasting window, methodology, prediction target, datasets used, and geographical resolution for each study. We also found that a large fraction of papers did not evaluate performance (25%), express uncertainty (50%), or state limitations (36%). To remedy some of these identified gaps, we recommend the adoption of the EPIFORGE 2020 model reporting guidelines and creating an information-sharing system that is suitable for fast-paced infectious disease outbreak science.

An expert judgment model to predict early stages of the COVID-19 pandemic in the United States.

From February to May 2020, experts in the modeling of infectious disease provided quantitative predictions and estimates of trends in the emerging COVID-19 pandemic in a series of 13 surveys. Data on existing transmission patterns were sparse when the pandemic began, but experts synthesized information available to them to provide quantitative, judgment-based assessments of the current and future state of the pandemic. We aggregated expert predictions into a single "linear pool" by taking an equally weighted average of their probabilistic statements. At a time when few computational models made public estimates or predictions about the pandemic, expert judgment provided (a) falsifiable predictions of short- and long-term pandemic outcomes related to reported COVID-19 cases, hospitalizations, and deaths, (b) estimates of latent viral transmission, and (c) counterfactual assessments of pandemic trajectories under different scenarios. The linear pool approach of aggregating expert predictions provided more consistently accurate predictions than any individual expert, although the predictive accuracy of a linear pool rarely provided the most accurate prediction. This work highlights the importance that an expert linear pool could play in flexibly assessing a wide array of risks early in future emerging outbreaks, especially in settings where available data cannot yet support data-driven computational modeling.

Outpatient healthcare personnel knowledge and attitudes towards infection prevention measures for protection from respiratory infections.

BACKGROUND: Healthcare personnel (HCP) knowledge and attitudes toward infection control measures are important determinants of practices that can protect them from transmission of infectious diseases. METHODS: Healthcare personnel were recruited from Emergency Departments and outpatient clinics at seven sites. They completed knowledge surveys at the beginning and attitude surveys at the beginning and end of each season of participation. Attitudes toward infection prevention and control measures, especially medical masks and N95 respirators, were compared. The proportion of participants who correctly identified all components of an infection control bundle for seven clinical scenarios was calculated. RESULTS: The proportion of participants in the medical mask group who reported at least one reason to avoid using medical masks fell from 88.5% on the pre-season survey to 39.6% on the post-season survey (odds ratio [OR] for preseason vs. postseason 0.11, 95% CI 0.10-0.14). Among those wearing N95 respirators, the proportion fell from 87.9% to 53.6% (OR 0.24, 95% CI 0.21-0.28). Participants correctly identified all components of the infection control bundle for 4.9% to 38.5% of scenarios. CONCLUSIONS: Attitudes toward medical masks and N95 respirators improved significantly between the beginning and end of each season. The proportion of HCP who correctly identified the infection control precautions needed for clinical scenarios was low, but it improved over successive years of participation in the study.

The United States COVID-19 Forecast Hub dataset.

Academic researchers, government agencies, industry groups, and individuals have produced forecasts at an unprecedented scale during the COVID-19 pandemic. To leverage these forecasts, the United States Centers for Disease Control and Prevention (CDC) partnered with an academic research lab at the University of Massachusetts Amherst to create the US COVID-19 Forecast Hub. Launched in April 2020, the Forecast Hub is a dataset with point and probabilistic forecasts of incident cases, incident hospitalizations, incident deaths, and cumulative deaths due to COVID-19 at county, state, and national, levels in the United States. Included forecasts represent a variety of modeling approaches, data sources, and assumptions regarding the spread of COVID-19. The goal of this dataset is to establish a standardized and comparable set of short-term forecasts from modeling teams. These data can be used to develop ensemble models, communicate forecasts to the public, create visualizations, compare models, and inform policies regarding COVID-19 mitigation. These open-source data are available via download from GitHub, through an online API, and through R packages.

Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States.

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policy makers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.

Collaborative Hubs: Making the Most of Predictive Epidemic Modeling

[...]models can vary in terms of what data they use, what they assume about transmission, and what analytic approach they use to produce projections. Because of this, relying on one model is dangerous because there is no guarantee that one model's choices and assumptions will yield an accurate prediction. In many fields, there is a long tradition of combining multiple models to mitigate this limitation by providing a single prediction that summarizes the view of the participating models.7 There has been a growing interest in using ensemble methodologies in epidemiology, with notable efforts in forecasting, risk prediction, causal inference, and decision-making.8-12 COORDINATION, COLLABORATION, AND EVALUATION A modeling "hub" is a consortium of research groups organized around a particular scientific challenge. The US COVID-19 Forecast Hub ensemble (including many component models) has struggled to produce accurate forecasts of cases and hospitalizations during periods of rapidly changing epidemic dynamics, such as the US peak of the winter wave in early 2021 or the rapid increases associated with the Delta variant in summer 2021 or in winter 2021-2022.3 Likewise, although longer-term projections from the COVID-19 Scenario Modeling Hub projected a Delta-associated resurgence in the United States, the ensemble significantly underestimated its speed and size, even though there were no clear deviations from scenario assumptions.13 However, even when projections are wrong, the hubs play a role in enhancing the scientific rigor and integrity of epidemic modeling. [...]operationally, there is value in developing procedures that harness the insights of a diverse network of scientists while guarding against groupthink and overconfidence.12 As researchers, system developers, and public health officials who have been deeply involved in the real-time operation of modeling hubs duringthe COVID-19 pandemic and prior epidemics, we believe the hub approach is a vital path forward for predictive disease modeling efforts.

Serological surveys to estimate cumulative incidence of SARS-CoV-2 infection in adults (Sero-MAss study), Massachusetts, July–August 2020: a mail-based cross-sectional study

ObjectivesTo estimate the seroprevalence of anti-SARS-CoV-2 IgG and IgM among Massachusetts residents and to better understand asymptomatic SARS-CoV-2 transmission during the summer of 2020.DesignMail-based cross-sectional survey.SettingMassachusetts, USA.ParticipantsPrimary sampling group: sample of undergraduate students at the University of Massachusetts, Amherst (n=548) and a member of their household (n=231).Secondary sampling group: sample of graduate students, faculty, librarians and staff (n=214) and one member of their household (n=78). All participants were residents of Massachusetts without prior COVID-19 diagnosis.Primary and secondary outcome measuresPrevalence of SARS-CoV-2 seropositivity. Association of seroprevalence with variables including age, gender, race, geographic region, occupation and symptoms.ResultsApproximately 27 000 persons were invited via email to assess eligibility. 1001 households were mailed dried blood spot sample kits, 762 returned blood samples for analysis. In the primary sample group, 36 individuals (4.6%) had IgG antibodies detected for an estimated weighted prevalence in this population of 5.3% (95% CI: 3.5 to 8.0). In the secondary sampling group, 10 participants (3.4%) had IgG antibodies detected for an estimated adjusted prevalence of 4.0% (95% CI: 2.2 to 7.4). No samples were IgM positive. No association was found in either group between seropositivity and self-reported work duties or customer-facing hours. In the primary sampling group, self-reported febrile illness since February 2020, male sex and minority race (Black or American Indian/Alaskan Native) were associated with seropositivity. No factors except geographic regions within the state were associated with evidence of prior SARS-CoV-2 infection in the secondary sampling group.ConclusionsThis study fills a critical gap in estimating the levels of subclinical and asymptomatic infection. Estimates can be used to calibrate models estimating levels of population immunity over time, and these data are critical for informing public health interventions and policy.

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines.

BACKGROUND: The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research. METHODS AND FINDINGS: We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies. CONCLUSIONS: These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

Modeling of Future COVID-19 Cases, Hospitalizations, and Deaths, by Vaccination Rates and Nonpharmaceutical Intervention Scenarios - United States, April-September 2021.

After a period of rapidly declining U.S. COVID-19 incidence during January-March 2021, increases occurred in several jurisdictions (1,2) despite the rapid rollout of a large-scale vaccination program. This increase coincided with the spread of more transmissible variants of SARS-CoV-2, the virus that causes COVID-19, including B.1.1.7 (1,3) and relaxation of COVID-19 prevention strategies such as those for businesses, large-scale gatherings, and educational activities. To provide long-term projections of potential trends in COVID-19 cases, hospitalizations, and deaths, COVID-19 Scenario Modeling Hub teams used a multiple-model approach comprising six models to assess the potential course of COVID-19 in the United States across four scenarios with different vaccination coverage rates and effectiveness estimates and strength and implementation of nonpharmaceutical interventions (NPIs) (public health policies, such as physical distancing and masking) over a 6-month period (April-September 2021) using data available through March 27, 2021 (4). Among the four scenarios, an accelerated decline in NPI adherence (which encapsulates NPI mandates and population behavior) was shown to undermine vaccination-related gains over the subsequent 2-3 months and, in combination with increased transmissibility of new variants, could lead to surges in cases, hospitalizations, and deaths. A sharp decline in cases was projected by July 2021, with a faster decline in the high-vaccination scenarios. High vaccination rates and compliance with public health prevention measures are essential to control the COVID-19 pandemic and to prevent surges in hospitalizations and deaths in the coming months.

The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research.

Forecasting has emerged as an important component of informed, data-driven decision-making in a wide array of fields. We introduce a new data model for probabilistic predictions that encompasses a wide range of forecasting settings. This framework clearly defines the constituent parts of a probabilistic forecast and proposes one approach for representing these data elements. The data model is implemented in Zoltar, a new software application that stores forecasts using the data model and provides standardized API access to the data. In one real-time case study, an instance of the Zoltar web application was used to store, provide access to, and evaluate real-time forecast data on the order of 108 rows, provided by over 40 international research teams from academia and industry making forecasts of the COVID-19 outbreak in the US. Tools and data infrastructure for probabilistic forecasts, such as those introduced here, will play an increasingly important role in ensuring that future forecasting research adheres to a strict set of rigorous and reproducible standards.

An expert judgment model to predict early stages of the COVID-19 outbreak in the United States.

During early stages of the COVID-19 pandemic, forecasts provided actionable information about disease transmission to public health decision-makers. Between February and May 2020, experts in infectious disease modeling made weekly predictions about the impact of the pandemic in the U.S. We aggregated these predictions into consensus predictions. In March and April 2020, experts predicted that the number of COVID-19 related deaths in the U.S. by the end of 2020 would be in the range of 150,000 to 250,000, with scenarios of near 1m deaths considered plausible. The wide range of possible future outcomes underscored the uncertainty surrounding the outbreak's trajectory. Experts' predictions of measurable short-term outcomes had varying levels of accuracy over the surveys but showed appropriate levels of uncertainty when aggregated. An expert consensus model can provide important insight early on in an emerging global catastrophe.

Estimation of Excess Deaths Associated With the COVID-19 Pandemic in the United States, March to May 2020.

Importance: Efforts to track the severity and public health impact of coronavirus disease 2019 (COVID-19) in the United States have been hampered by state-level differences in diagnostic test availability, differing strategies for prioritization of individuals for testing, and delays between testing and reporting. Evaluating unexplained increases in deaths due to all causes or attributed to nonspecific outcomes, such as pneumonia and influenza, can provide a more complete picture of the burden of COVID-19. Objective: To estimate the burden of all deaths related to COVID-19 in the United States from March to May 2020. Design, Setting, and Population: This observational study evaluated the numbers of US deaths from any cause and deaths from pneumonia, influenza, and/or COVID-19 from March 1 through May 30, 2020, using public data of the entire US population from the National Center for Health Statistics (NCHS). These numbers were compared with those from the same period of previous years. All data analyzed were accessed on June 12, 2020. Main Outcomes and Measures: Increases in weekly deaths due to any cause or deaths due to pneumonia/influenza/COVID-19 above a baseline, which was adjusted for time of year, influenza activity, and reporting delays. These estimates were compared with reported deaths attributed to COVID-19 and with testing data. Results: There were approximately 781 000 total deaths in the United States from March 1 to May 30, 2020, representing 122 300 (95% prediction interval, 116 800-127 000) more deaths than would typically be expected at that time of year. There were 95 235 reported deaths officially attributed to COVID-19 from March 1 to May 30, 2020. The number of excess all-cause deaths was 28% higher than the official tally of COVID-19-reported deaths during that period. In several states, these deaths occurred before increases in the availability of COVID-19 diagnostic tests and were not counted in official COVID-19 death records. There was substantial variability between states in the difference between official COVID-19 deaths and the estimated burden of excess deaths. Conclusions and Relevance: Excess deaths provide an estimate of the full COVID-19 burden and indicate that official tallies likely undercount deaths due to the virus. The mortality burden and the completeness of the tallies vary markedly between states.

The Incubation Period of Coronavirus Disease 2019 (COVID-19) From Publicly Reported Confirmed Cases: Estimation and Application.

BACKGROUND: A novel human coronavirus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was identified in China in December 2019. There is limited support for many of its key epidemiologic features, including the incubation period for clinical disease (coronavirus disease 2019 [COVID-19]), which has important implications for surveillance and control activities. OBJECTIVE: To estimate the length of the incubation period of COVID-19 and describe its public health implications. DESIGN: Pooled analysis of confirmed COVID-19 cases reported between 4 January 2020 and 24 February 2020. SETTING: News reports and press releases from 50 provinces, regions, and countries outside Wuhan, Hubei province, China. PARTICIPANTS: Persons with confirmed SARS-CoV-2 infection outside Hubei province, China. MEASUREMENTS: Patient demographic characteristics and dates and times of possible exposure, symptom onset, fever onset, and hospitalization. RESULTS: There were 181 confirmed cases with identifiable exposure and symptom onset windows to estimate the incubation period of COVID-19. The median incubation period was estimated to be 5.1 days (95% CI, 4.5 to 5.8 days), and 97.5% of those who develop symptoms will do so within 11.5 days (CI, 8.2 to 15.6 days) of infection. These estimates imply that, under conservative assumptions, 101 out of every 10 000 cases (99th percentile, 482) will develop symptoms after 14 days of active monitoring or quarantine. LIMITATION: Publicly reported cases may overrepresent severe cases, the incubation period for which may differ from that of mild cases. CONCLUSION: This work provides additional evidence for a median incubation period for COVID-19 of approximately 5 days, similar to SARS. Our results support current proposals for the length of quarantine or active monitoring of persons potentially exposed to SARS-CoV-2, although longer monitoring periods might be justified in extreme cases. PRIMARY FUNDING SOURCE: U.S. Centers for Disease Control and Prevention, National Institute of Allergy and Infectious Diseases, National Institute of General Medical Sciences, and Alexander von Humboldt Foundation.